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Introduction 


Many science educators have made efforts to develop and implement 
pre- or in-service teacher programs to improve science teaching and learning 
(Supovitz & Turner, 2000). However, some educators also have criticized that 
traditional teacher programs are inadequate and ineffective in improving 
science teaching and learning in schools (Driel, Beijaard, & Verloop, 2001; 
Little 1993; Russell, McPherson, & Martin, 2001). For instance, Zeichner and 
Tabachnick (1981, p. 7) pointed out that the impact of teacher training at the 
college level is “washed out” when new teachers enter the classroom, and 
Hobson et al. (2008, p. 113) mentioned that “... many student teachers ... 
(agree with) that they learn more about teaching in schools rather than uni- 
versity classes.” Joram and Gabriele (1998) also reported that educational psy- 
chology courses at the college level had little impact on pre-service students’ 
views of teaching and learning. For example, in the prerequisite test before 
taking the educational psychology course, pre-service students responded 
that learning is “absorbing information” (36%) and “transfer” (23%) but none 
responded that learning is “change or reorganization of new knowledge.’ 
Even after taking the educational psychology course, students still replied 
that learning is “absorbing information” (28%) and “transfer” (42%). Only 4% 
answered that learning is “change or reorganization takes place.’ Smylie (1989) 
found that among 14 professional development resources, teachers ranked 
their undergraduate education courses as 13th, and ranked the in-service 
program provided by their own school districts as 14th. 

In an effort to improve teacher learning, science educators have aimed 
to expose teachers to effective, evidence-based, and up-to-date educational 
theories and teaching strategies through many teacher-training programs, 
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Abstract. An observational instrument 
called the KTOP (Korean Teaching Observa- 
tion Protocol) was developed to analyze 
science teaching and to identify the compo- 
nents of teaching that need improvement. 
To diminish the gap between theory and 
practice in science teaching, the method 
for developing the KTOP was structured as 
consisting of three steps: the first version 

of the KTOP was developed through direct 
observation of science classes at the first 
step, revised through literature review 

with consideration of the Korean national 
curriculum and comparison with other 
observational instruments in the second 
step, and finalized after application to ac- 
tual teaching context. As a result, the KTOP 
was determined to consist of 30 items of 10 
sub-categories in 4 main categories. The 
content was validated through 48 teachers’ 
responses to the questions asking if each 
indicator of the KTOP is worthwhile to be 
considered in their teaching. Reliability 
was obtained by agreement (72~90%) 

and correlation (r=0.90) among observers’ 
KTOP scores. Finally, the cases of applying 
the KTOP to improve science classes and 

to develop teachers’ expertise through an 
in-service program were described. 
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which encourage teachers to apply what they have learned to their teaching practice. However, if such effective 
teacher-training programs are not based on and closely connected to everyday practical teaching, these programs 
will be ineffective in improving teaching practice (Lieberman, 1995; Supovitz & Turner, 2000). According to Put- 
nam & Borko (2000, p. 6), teachers often complain that professional development learning experiences are not 
in line with everyday teaching situations; as a result, the learning experiences do not have meaningful effects on 
their teaching practice. This disconnect may be because many teacher programs are too theoretical and abstract 
(Loughran, Berry, & Mulhall, 2006; Vick, 2006) without detailed consideration of real teaching situations in schools 
(Driel et al., 2001; Korthagen, 2007; Wallace & Louden, 1992). Wallace and Louden (1992, p. 517) concluded that 
“The problem... (is) the assumption that teaching is a matter of applying a set of generalizable skills to given situ- 
ations and that the role of the teacher is to simply choose which skills to apply.’ 

The literature suggests that teacher-training programs should be based on actual science teaching practices, 
and also should be designed to improve science classes in practice (Guskey, 2003; Hascher, Cocard, & Moser, 2004; 
Hiebert, Gallimore, & Stigler, 2002; Nielsen, 2012). Guskey (2003, p. 749), based on a review of about 13 lists of 
the features of effective teacher development, reported that the majority of the lists emphasizes a school-based 
teacher program. Lieberman (1995, p. 67) noted that “What everyone [teachers] appears to want [are learning ex- 
periences]... that engage them in experiencing, creating, and solving real problems, using their own experiences.” 
Rogers et al. (2007) also found that many teachers want“classroom application (including classroom resources)” for 
more effective professional development. In addition, Supovitz and Turner (2000) stressed teacher engagement in 
concrete teaching tasks as one of the six elements for “high quality professional development.’ 

A more practice training approach is demanded for effective teacher professional development because real 
teaching is complex and unpredictable (Driel et al., 2001; Hoban, 2005; Korthagen, 2007; Wallace & Louden, 1992). 
Teaching “is influenced by the interaction of many elements such as the curriculum, the context, and how students 
respond to instruction...” (Hoban, 2005, p. 9). Therefore, to analyze and improve science teaching, we should observe 
teaching in classrooms directly and consider the teaching context, including classroom conditions and facilities, 
students’ attitudes, interests and cognitive levels, teachers’ teaching experiences, belief and style, and so on. 

A single teaching theory or strategy is not guaranteed to be effective in every teaching situation, even though 
it may have been effective in a specific teaching situation (Trianou, 2007, p. 37). Therefore, ideas for improvement or 
guidance should be flexible and context-dependent, based on analysis of direct observations of science teaching 
under investigation. To meet this criterion, researchers have decided to develop a practical and effective instru- 
ment to analyze teaching practices for improvement through observation, and to suggest concrete improvement 
ideas and methods that can be used in everyday teaching situations. The literature indicated the difficulty of 
developing such an instrument of analyzing and evaluating science classes (Linn, 2000; Tittle & Pape, 1996). Some 
observational instruments have already been used, such as RTOP (The Reformed Teaching Observation Protocol) 
designed by the Evaluation Facilitation Group of the Arizona Collaborative for Excellence in the Preparation of 
Teachers (ACEPT) (Sawada et al., 2002), or OTOP (The Reformed Teaching Observation Protocol) designed by the 
Evaluation Facilitation Group of the Oregon Collaborative for Excellence in the Preparation of Teachers (OCEPT) 
(Wainwright, Flick, & Morrell, 2003). 

These tools (RTOP & OTOP) have been used for various goals. For instance, the RTOP can be used to understand 
the features of science teaching in the classroom and the laboratory. Campbell (2010) used the RTOP to investigate 
the similarities and differences between Korean and US science classrooms for grades 9-12. The instrument showed 
no significant differences in the RTOP scores between the two countries, but US teachers showed a higher score 
than Korean teachers in nine items of the RTOP (e.g., Item 21: “active participation of students was encouraged 
and valued”). 

The RTOP is also a useful tool for evaluating the teaching profession and the changes within it. For instance, 
Akkus and Hand (2011) evaluated the impact of a pedagogical model on mathematics classrooms, and Amerin- 
Beardsley and Popp (2012) showed the RTOP to be a formative evaluation tool of science and mathematics teachers 
through peer observation. 

Furthermore, RTOP scores can be used to predict students’ achievement. For instance, Blanchard et al. (2010) 
observed that, when applying guided inquiry-based instruction to middle and high school students, students whose 
teachers had high RTOP scores attained higher scores than other students whose teachers had low RTOP scores, and 
Jong, Pedulla, Salomon-Fernandez, & Cochran-Smith (2010) reported a statistically significant correlation between 
students’ mathematics test scores and teachers’ RTOP scores (r = 0.56). Park (2006; 2008) also employed the OTOP 
tools to profile explicit teaching strategies displayed by experienced teachers in elementary and secondary level 
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in the context of scientific inquiry to realize the commonalities and difference between two groups. Wainwright et 
al. (2003) also employed OTOP as an assessing tool to analyze instructional practices in standards-based teaching. 
Flick, Sadri, Morrell, & Wainwright (2009) employed the OTOP instrument to profile faculties’ instructional practices 
to improve their teaching practices in the field of math and science at universities of Oregon. 

Likewise, RTOP as well as OTOP can be used for our purpose of improving science classroom teaching. How- 
ever, as stressed earlier, analysis and improvement effort for science teaching should be context-based. Therefore, 
we aim to develop a Korean version of teaching observational protocols. The new instrument has been named 
KTOP (Korean Teaching Observation Protocol) and shares the goals of RTOP and OTOP in its development and 
employment in researches. 

Korea has a national science curriculum that describes the goals, teaching plans, and assessment as well as 
contents to be learned in each grade from grades 3 to 12. Therefore, Korean science teachers need to consider the 
national curriculum when teaching science in schools. 

In Korea, to become a public science teacher, students who graduate from the college of education for sec- 
ondary education must pass the national teacher recruitment examination. Because this examination is highly 
competitive (for example, 12 to 1 in biology, 8 to 1 in physics, 11 to 1 in chemistry, and 7 to 1 in earth science in 
the year of 2013), marking the examination is very rigorous. Starting in 2014, the national recruitment examina- 
tion format consists of two steps: 1) the essay test as a type of written test consisting of short answers, descriptive 
writing as well as a logical one, and 2) actual teaching presentation and interview. All candidates who have earned 
their teacher certificates and registered for national teacher recruitment examination can take the essay test, which 
evaluates knowledge in pedagogy as well as science content in the first stage of examination. Then 2 times the 
number of final teacher recruits at public schools take the 2"? examination, where two kinds of assessments are 
conducted. In this examination, the first screened teacher candidates show up at their own provinces and take 
the second examination. In the first assessment, the superintendents or provincial officers of education interview 
teacher candidates about the teacher image, philosophy of teaching, and personal character as teachers with the 
same protocols. Teacher candidates are also interviewed to answer what they would do if they were in a situation 
of educational risk. In the second assessment, the teacher candidates’ abilities of developing a lesson plan and 
demonstrating actual teaching performance are assessed under a given condition. Teacher candidates teaching 
from their own lesson plan can be evaluated by the observational assessing protocol instrument, in which case we 
can use KTOP in this practice examination to evaluate actual science teaching performance. Because we aimed to 
develop an instrumental tool specialized and appropriate for improving science teaching and learning, the KTOP 
includes particular science-specific features such as “encouraging scientific thinking skills” or “teaching various 
types of scientific inquiry.’ 

Finally, the efforts by researchers in this study to evaluate and improve science teaching in practice, with 
consideration of various everyday teaching contexts, do not mean that learning educational theories and effective 
teaching strategies is unnecessary. Rather, practical improvement of science teaching should be theory-based and 
research-based (Bednar, Cunningham, Duffy, & Perry, 1995; Hascher et al., 2004; Kearns et al., 2010). Therefore, when 
developing observational instruments, the research team considered various educational and cognitive theories 
as well as educational research results. For example, a reliable set of observable teaching practices was considered 
as an essential factor (Oh et al., 2008; Rosenshine & Stevens, 1986), and the students’ abilities of developing argu- 
mentation were defined as those of covering purposeful practice involving reflection on how students differentiate 
theory from evidence and vice versa (Kuhn, 1992, 2007; Park, 2008). The significance of explicit teaching practices 
in modeling and thinking through argumentation was also emphasized (Kim, Mun, Park, & Lim, 2010; Krajcik et 
al, 1998; Kwak, 2010; Pressely, Hogan, Wharton-McDonald, Mistretta, & Ettenberger, 1996) as one of the pivotal 
factors in developing the KTOP. 


Research Objectives 
Based on the above background, the research objectives are as follows: (1) to develop the KTOP (Korean Teach- 
ing Observation Protocol) to detect the parts needed for improvement in actual science teaching in the classroom 


or laboratory, (2) to describe the basic goals, the process of development, and validity and reliability of the KTOP, 
and (3) to introduce the some applications of the KTOP. 
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Development of the KTOP 
Basic Goal and Components of the KTOP 


The basic purpose of the KTOP is to use it to improve science classes by observing the science classes directly 
and analyzing areas for improvement. To this end, the KTOP includes counter-examples of areas of improvement 
and examples of actual improved teaching situations (Table 1). All counter-examples and examples for each item 
of the KTOP were collected from actual observations of science classes and from efforts for improving science 
classes through researcher-teacher cooperation. More detailed explanations of how to use the KTOP for improving 
science classes will be given in the later section, “Application of the KTOP: Applying to POCOM (Practical On-Site 
Cooperation Model). 


Table 1. |§Aninstance of aKTOP item consisting of four components. 
KTOP item Students are on task actively in class 


a - There are activities, presentation, discussion and games in groups where students construct and exchange 

Operational definition at hee ee 
their opinions rather than passively listening to teachers’ lectures. 

- The lesson consists of teacher-centered lectures during which students copy notes from the board or provide 

Counter-examples simple answers. 

- There is group work, but only a few students lead the activity or lessons, and others participate passively. 


- Teacher makes students think about the core concept and show related phenomenon. Then, students write the 
observations on activity sheets and compare them with peers in groups. 

- A teacher explains why such a phenomenon happened instead of presenting a direct explanation. Students 
exchange opinions with peers in groups and answer the teacher’s questions. 


Examples 


Developing Process of the KTOP 


To develop an observational tool which can be used to identify the aspects of improvement for better science 
teaching, the ‘direct observation method’ (Dragnidis & Mentzas, 2006; Parry, 1996) and the ‘generic model overlay 
method’ (Dubois, 1993) were applied. These methods have been used to identify components of competency 
required to perform and complete a given mission or task in more successful and effective ways in the areas of 
management, psychology, education, and so on. 

The ‘direct observation method’ can be used to extract and code characteristic behaviors when observa- 
tion subjects perform a given task, and to generate a tentative list of competencies. For instance, Draganidis and 
Mentzas (2006, p. 56) noted that, “A competency model is a list of competencies which are derived from observing 
satisfactory or exceptional employee performance for a specific occupation.” In this study, direct observation of 
science teaching in everyday context was conducted to generate the first version of the KTOP. This method was 
used because, as noted in the introduction section, many educators have emphasized that improving teachers’ 
expertise or actual teaching in the classroom should be based on everyday teaching situation considering the 
teachers’ teaching attitude or habit, students’ interest or cognitive level, classroom atmosphere, and so on (e.g., 
Hascher et al., 2004; Hoban, 2005; Putnam & Borko, 2000; Korthagen 2007). 

“Isolating the characteristics of exemplary performers is achieved by observing them... competencies charac- 
teristic of exemplary performers only become the basis for complete competency model development.’ (Rothwell 
& Lindholm, 1999, p. 98). This means that the initial model needs revision. Therefore, we adopted the ‘generic model 
overlay method; in which the prior existing model is revised through various processes, such as comparison with 
other models, consideration of the developer's own goals, and so on. In this study, the first version of the KTOP 
was revised based on literature review, analysis of the Korean national science curriculum, and comparison with 
existing other observational tools (RTOP and OTOP). This is because ‘practice should be based on the theory’ as 
stressed by many educators (e.g., Bednar et al., 1995; Kearns et al., 2010; Hascher et al., 2004) 

Using these two approaches, it is expected that the gap between theory and practice can be minimized. Finally, 
the KTOP was confirmed and revised again by applying it to actual science teaching to increase its concreteness 
and feasibility. 
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As a result, our developmental method for the KTOP can be structured as shown in Figure 1, consisting of 
three steps: bottom-up approach, top-down approach and field test. 


3rd STEP 


Field Test 
- applying the 2nd version of the KTOP to actual science classes 
- finalizing the KTOP consisting of 30 indicators in total out of 10 sub-categories in 4 main categories 










2nd STEP 
Top-down Approach 


- reviewing relevant literature 

- analyzing Korean national science curriculum 

- comparing our instrument with RTOP and OTOP 

- suggesting the 2nd version of the KTOP consisting 26 indicators with 4 main categories 





ist STEP 









Bottom-up Approach 
- observing science classes in everyday context 

- extracting instances indicating 'good' and 'bad' teaching from the observation 
- summarizing and categorizing the extracted instances 

- suggesting the first version of the KTOP consiting of 12 indicators 





Figure1: | Developmental process of the KTOP through 3 steps. 


First step 


In the bottom-up approach, we (professionals and experts in science education) observed a science class 
in everyday context, and discussed the parts that needed improvement with an open mind, that is, without any 
pre-determined observational tool. As professional science educators, we teach college students in pre-service 
courses at the university; develop and implement in-service programs to develop teachers’ expertise and to help 
science teachers’ actual teaching in schools; participate in reforming the national science curriculum; and publish 
science text books. And we also have experience teaching science at the junior or secondary high school level, 
as science teachers before working as science educators at the university. Therefore, we have intuition, practical 
experience, and professional expertise on what good science teaching is in schools. 

After observation of two typical science classes (earth science classroom and chemistry laboratory teaching), 
for the aspects of improvement in science teaching and learning were determined. Through this process, instances 
that indicated “good” or “to be improved” science teaching and learning were extracted as shown in Table 2. 


Table 2. Summary of features of the observed science classes using the bottom-up approach. 


Area to be improved 


Category Observation or good indicator 
The overall direc- Main featuresof  - Teacher read and explained textbook, told students to - Teacher needs to consider constructivist 
tion/ philosophy whole process underline parts of the text and asked questions requir- teaching and learning 
of learning ing short/exact answers. Finally, teacher summarized 


the main content on the blackboard. 


- Teaching consisted of providing and explaining sepa- - Teacher needs to provide a coherent explana- 
rate scientific concepts and information. tion based on theory, model, or big idea. 
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Category 


Cognition of learningMeta 


-cognitive 

awareness 
Teacher’s Practices for 
PCK explaining 


Learning practices Students’ behavior 
/practices 


Teacher's teaching 
strategy 


Interaction between 
teacher and 
students or among 
students 


Inquiry activity 


Assessment 
learning 


Assessing students’ 


Observation 


- Teacher checked what students learned and knew. 
- A teacher encouraged students to consider how/why 
questions while conducting experiments. 


- Teacher confuses the relationship between the theory 
and evidence. That is, he/she used the theory instead of 
evidence supporting natural phenomena. 


- Teacher failed to help students change their misconcep- 
tion identified in the teaching. 


- Teacher sometimes taught science concepts wrongly or 
incompletely, or skipped important concepts. 


- Teacher missed the link between big ideas and small 
concepts. 


- There were no student-centered activities except 
answering short answers, reading the textbook, and 
underlining phrases in the textbook. 

- In the final step of the lesson, students summarized 
what they learned using a simple but interesting game. 


- There were no specific teaching strategies except 
asking questions, reading from the book, and underlining 
important phrases in the book. 

- Teacher elicited mainly closed answers. 


- The only interaction was asking and answering the 
questions. 

- Teacher ignored students’ unexpected questions and 
responses. 


- Students did not carry out the activities to develop vari- 
ous inquiry skills*. 
- Teacher did not use appropriate demonstrations. 


- Teacher gave score to students who answered the 
questions, and the teacher tried to give all students the 
opportunity to answer the questions. 


ISSN 1648-3898 


Area to be improved 
or good indicator 


- Good indicator 


- Good indicator 


- Teacher needs to understand the NOS (e.g., 
difference between theory and evidence). 


- Teacher needs to consider a conceptual change 
model. 


- Teacher needs to understand scientific concepts 
thoroughly. 


- Teacher needs to link small concepts with big 
ideas. 


- Teacher needs to encourage students to 
participate in their own learning through various 
activities. 

- Good indicator 


- When reading the textbook, the teacher needs 
to use a variety of reading and summarizing 
strategies. 

- Teacher needs to use open questions (for 
creativity). 


- Teacher needs to encourage students to think, 
present, compare, and discuss their own ideas. 
- Teacher should deal effectively with students’ 
various responses. 


- Teacher needs to include “to differentiate,” “to 
interpret the data” or “to think inductively” as teach- 
ing goals. 

- Interesting demonstration is preferred. 


- Good indicator for formative assessment and 
active participation. 


*Inquiry skills in KTOP include those of integrated process skills as well as basic one in SAPA (Science: A Process Approach) (Turpin & 


Cage, 2004). 


The “good indicators” in the last column of the Table 2 needs to be continually encouraged and the “to be 
improved” aspects need to be considered for improving science classes. As a result, based on Table 2, the following 
12 items were determined as basic components of the first version of the KTOP. 


1. Constructivist teaching and learning need to be encouraged. 


~ 


tant. 


oe a ae 


students’ interest, motivation or active participation. 
9. Open questions are recommended to develop scientific thinking skills or creativity. 


Various connections between small concepts and big idea, theory and evidence and so on are impor- 


Metacognitive awareness during classroom learning and experiments is necessary. 

Teaching and learning science should be based on correct NOS understanding. 

Considering students’ misconceptions and the change of them are required. 

Correct understanding and exact explanation of scientific concepts are essential. 

Various teaching strategies to encourage students’ active participation are necessary. 

Various learning activities, including games, demonstrations, and so on, are important to improve 
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10. Presenting, comparing, and discussing students’ own ideas, as well as asking and answering the ques- 
tions, are important for active communication. 

11. Various types of inquiry activities (inductive, deductive, and abductive) that require various types of 
inquiry skills (integrated as well as basic process skills in SAPA) are recommended. 

12. Appropriate and effective assessment is necessary. 


Among the above items, some items can be extended for further comprehensiveness: for instance, the second 
item can include other connections such as the link between scientific concepts and everyday experience or the 
link between today’s learning concepts and students’ prior knowledge. The initial item can be subdivided even 
further; for instance, the eighth item can be divided into activity for interest/motivation and activity for students’ 
participation. This process was conducted using the following approaches. 


Second step 


The second step, that is, the top-down approach, revises the first version of the KTOP. First, we reviewed the 
science education literature about the basic philosophy and fundamental goals of science teaching and learning, 
and various teaching elements or conditions including teaching strategies, learning activities, assessment tools, 
and so on. Table 3 indicates a brief description of the literature review that was done to obtain the theoretical 
background for developing the KTOP. 


Table 3. Features stressed in the literature for “better” teaching. 


Source Features 


Abrams (1998), Cimer (2007), For effective teaching, they stressed (1) students’ existing ideas and conceptions, (2) application of new concepts 
Porter and Brophy (1988) or skills to new contexts, (3) students’ active participation in lessons, (4) inquiry and cooperative learning, and (5) 
continuous assessment with feedback. 


Burry-Stock (1995) He developed a rubric for constructivist teaching and learning, consisting of 4 categories and 18 items. The four 
categories are (1) learning from a constructivist perspective, (2) pedagogy related to student understanding, (3) 
strategies based on interactions with students, and (4) teacher knowledge of subject matter. 


NRC (1996) It stresses a shift from traditional pedagogy to more reformed one. It describes which aspects need to be ‘less 
emphasized (e.g., learning science by lecture and reading)’ and which aspects need to be ‘more emphasized (e.g., 
collegial and collaborative learning)’. 


Supovitz and Turner (2000) They emphasized (1) concrete experience before introducing an abstract concept, (2) connection of science with 
other disciplines, (3) use of computers, (4) various types of questions to evaluate students’ understanding, (5) 
inquiry activity in groups, and (6) development of conceptual understanding. 


Weiss, Banilower, Crawford, They suggested sample indicators for assessing the quality of the lesson, consisting 4 categories with 21 items. 

and Overstreet (2003) The four categories are (1) design of lesson (e.g., design encouraging a collaborative approach), (2) implementation 
(e.g., adjusting instruction according to students’ level of understanding), (3) science content (e.g., accurate content 
information, and connection of science to other disciplines), and (4) classroom culture (e.g., active participation of 
all students). 


Tigellar, Dolmas, Wolfhagen, — For teaching competencies in higher education, they identified 61 items (e.g., a teacher can activate students, as- 
and van der Vleuten (2004) sess students’ learning results, and so on) categorized into 7 domains (e.g., expert on content knowledge, facilitator 
of learning processes as an evaluator, and so on). 


Schraw, Crippen, and Hartley —_ For constructivist-oriented instruction in the classroom, they stressed (1) inquiry based learning, (2) collaborative 
(2006) support, (3) improvement of problem-solving and critical thinking, (4) support to help students to construct mental 
models and experience conceptual change, (5) use of technology, and (6) impact of students’ and teachers’ beliefs. 


Kim et al. (2010), Oh et al. They stressed the teacher’s expertise in (2) organizing science content, (2) evaluating students’ understanding, (3) 
(2008) demonstrating appropriate PCK, and (4) preparing materials appropriate to students’ levels. 

MCEECDYA They provided a standard for professional teaching consisting of 3 domains; professional knowledge, professional 
(2010) practice, and professional engagement. Among them, professional practice, which relates to our concern, specifi- 


cally, consists of 3 categories with 18 items. 


As mentioned, the areas needing improvement should adhere to the standards of teaching and learning science 
described in the science curriculum. According to the Korean national science curriculum, as well as the concepts 
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and inquiry activities to be taught, the strategies of teaching and learning science are described as follows: how to 
plan lessons, what materials to prepare and how to use them, how to teach, how to guide experimentation, how 
to support teaching and learning, and lastly how to evaluate learning related to science. Therefore, when revising 
the KTOP, we considered how we could help science teachers equip themselves with professional strategies of 
teaching and learning science as described in the Korean national science curriculum. 

In the second step, the RTOP and the OTOP were also considered to revise the first version of the KTOP. The 
RTOP consists of 25-item classroom observation protocols for“reformed” teaching, emphasizing (a) standards based; 
(b) inquiry-oriented; and (c) student-centered teaching and learning (Sawada et al., 2002). The OTOP consists of 
ten categories of concrete and measurable teachers’ practices, along with students’ responses (Wainwright et al., 
2003). 

In RTOP, some items cannot be measured as explicit teaching practices. For example, one item of teaching, 
“teacher had a solid grasp of the subject matter content in the lesson” from RTOP, is not explicit in measuring how 
much the teacher conveys the concept appropriately or accurately, because “knowing” is different from “teach- 


n" 


ing: 

In the comparison of the RTOP with OTOP it was found that the OTOP consists of ten categories of concrete 
and measurable teachers’ practices with students’ responses, unlike the RTOP, where only teachers’ teaching strate- 
gies are indicated. Therefore, we determined that the students’ activities or responses, as well as teachers’ practices, 
needed to be encouraged in the KTOP. 

Comparing the observation of actual science teaching (in the bottom-up approach) and literature review (in 
the top-down approach) with the RTOP and OTOP, it was found that the following characteristics were excluded 
in the RTOP and the OTOP; 


1. Teaching the nature of science; 

2. Implementing various types of inquiry activities (inductive, deductive, or abductive inauiry) as well as 
a confirmation activity; 

3. Checking the preparation and clean-up of the lab materials, in addition to lab safety; 

4. Promoting students’ interest/motivation in their learning; and 

5. Employing appropriate assessments of students’ learning. 


Finally, to make the KTOP a tool explicitly specialized for science teaching and learning, rather than for 
teaching and learning of all subjects like the RTOP and the OTOP, we added the following statement: “Students 
understand the inquiry process meta-cognitively rather than blindly follow a procedure,” Here, an inquiry skill or 
process through inquiry activities refer to those of inquiry processes defined in SAPA, consisting integrated inquiry 
processes as well as basic ones as defined in Table 2.“Teacher encourages students to develop or employ scientific 
thinking (deductive, inductive, and abductive thinking) during the class” or “Students apply what they learned to 
various contexts (daily life, engineering/technology or environmental situations)” 

Through this top-down process (literature review and consideration of the Korean national science curriculum 
and the RTOP and the OTOP), the initial 12 items are revised into 26 items with 4 main categories (philosophy, 
learning goals, activities and assessment) and 10 sub-categories (constructivism, the nature of science, conceptual 
understanding, inquiry, development of thinking, interest and motivation, connection, teaching strategies and 
materials, cooperation and communication, assessment). 


Third step 


In the third step, the revised version of the KTOP was applied to actual science teaching in a field test for 
validation. Here, we checked whether we could profile teachers’ and students’ practices of science teaching and 
learning easily by using the indicators from the KTOP. For example, in a chemistry lab class, we could easily observe 
students’participation in a teacher’s demonstration of a flame reaction with different chemical materials; however, 
we were not sure if students were on task; that is, it was difficult to interpret whether students were participating 
in their learning even though students were watching a teacher’s demonstration. If students had been interact- 
ing with the teacher through questions and answers during the demonstration, then we could have interpreted 
their participation as active learning. Therefore, the term of “participation” was revised to “on-task” to indicate that 
participation must be active, not passive. 
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In another example of a chemistry lab class, more definitions were added to describe what could be observed 
during the lab class. For example, it was found that teachers needed to check students’ prior inquiry skills before 
conducting the inquiry, and to encourage students to compare the results with hypothesis or prediction in the 


final step of the experiment. 


In a physics classroom, it was observed that the teacher introduced various characteristics of “light” only 
verbally. In this case, it was confirmed that using materials or demonstrations is essential to making abstract 


concepts concrete. 


As aresult, after the revision through the field test, we determine 30 items of four main categories (philosophy, 
learning objective, practices, and assessment) and ten sub-categories, as shown in Table 4. Here, the prior five catego- 
ries (Table 2) were reduced into four: philosophy, goals of learning, activities, and assessment. The prior categories 
of goal of learning and goal of teaching in Table 2 were combined into one, learning objective, in Table 4. 


Table 4. KTOP (Korean Teaching Observation Protocol). 


Category 


Philosophy 1. Constructivism 


Goals of learning 2. Nature of Science 


3. Concept 
Understanding 


4. Scientific Inquiry 


5. Scientific 
Thinking 
Development 


6. Motivation 
and Interest 


Item 


1.1 Students are on task actively in class. 
1.2 Students realize the objectives and products of class activities metacognitively. 


1.3 Teacher, as a supporter, facilitates students’ learning, not directs it. 


2.1 Teacher helped students understand the nature of science. 


3.1 Teacher encourages students to reflect on their prior knowledge, and employs conceptual change 
model if there was any misconception. 


3.2 Teacher explains core concepts appropriately and accurately. 


3.3 Teachers make difficult and abstract concept (including graphs, code, symbol, etc.) more under- 
standable and easy. 


3.4 Students model and construct their own understanding through active exploration and inference 
activities. 


3.5 Students reflect on their own or other students’ understanding, metacognitively. 


3.6 Students extend their basic understanding about concepts to a more complicated and wider 
scope. 


4.1 All materials are prepared before lab activity and students clean up after lab activity. 
4.2 Teacher provides lab safety instructions to students before and during experimentation. 


4.3 Teacher checks students’ existing inquiry skills and provides appropriate instruction for revising 
them if necessary. 


4.4 Students realize the inquiry process meta-cognitively rather than blindly follow a procedure. 


4.5 Teacher instructs students to develop inquiry skills in more detail if necessary, such as skills of 
observation, variables control, or hypothesizing. 


4.6 Students evaluate the experimental results by comparing them with the hypothesis and inferring 
reasons for errors. 


4.7 Students carry out various types of inquiry activities, such as inductive or hypothesis-deductive 
activities. 


5.1 Teacher encourages students to develop or employ scientific thinking (deductive, inductive, and 
abductive thinking) in class. 


5.2 Students have a chance to develop their claims by discussion or argumentation with supporting 
evidence. 


5.3 Teacher encourages creative thinking through various activities. 


6.1 Teacher motivates/interests students in their learning. 
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Category Item 


Activities 7. Connection 7.1 Teacher integrates content with other classes or interdisciplinary curricula. 


7.2 Students apply what they learned to various contexts (daily life, engineering/technology, or 
environmental situation). 


8. Learning 8.1 Students use various learning strategies such as analogy, concept mapping, and so on. 
Seiad 8.2 Various appropriate media such as computer, graphics, drawings, video, demonstration, and so 
oe on are used. 
8.3 Students participate actively in various learning activities such as worksheet activities, games or 
role plays. 
9. Cooperation 9.1 Teacher encourages students to be cooperative in learning through group activities. 


and Communication 9.2 Students communicate with teachers or with one another through discussion and presentation. 


9.3 Students ask and answer questions actively, and the teacher responds to spontaneous questions 
appropriately. 


Assessment 10. Assessing 10 Teachers assess if learning objectives are met during the class. 


Description of 4 Main Categories of the KTOP 
Philosophy 


In the KTOP, constructivism was adopted as a fundamental educational philosophy in science teaching and 
learning. Therefore, students’ active learning, students’ metacognitive awareness to their own learning, and the 
teacher’s role as a supporter for students’ learning were emphasized. 


Basic Goals 


Science teaching should help students to achieve the basic goals pursued in a science class. According to the 
Korean national science curric13ulum, the following five aspects have been emphasized in teaching and learning 
science in schools: (1) concept, (2) scientific inquiry, (3) NOS, (4) scientific thinking development, and (5) motiva- 
tion and interest. Understanding scientific concepts and improving scientific inquiry skills are the fundamental 
and common goals in all science curricula. The “Physics I” curriculum includes such goals as“... to improve inquiry 
ability based on the nature of science...” (Ministry of Education, Science and Technology [MEST], 2011, p. 85), and 
goals such as”... to improve creative and integrative thinking ability...” (MEST, 2011, p. 2) are included in the sci- 
ence curriculum for middle school students. The Korean science curriculum also stresses motivation and interest 
for students of all grade levels. 


Activities 


Science teaching needs to be planned and organized to encourage students to participate in various active 
activities. Therefore, it was emphasized that students can connect what they learned in science class with other 
subjects or everyday life through active cooperation and communication using various learning strategies and 
materials. Regarding “7. Connection” in Table 3, many science curriculum has stressed the understanding of the 
relationship among science, technology and society; problem solving ability in everyday contexts; and recently, 
interdisciplinary integration such as “STEM.” The “Teaching and learning methods” standard, which is included in 
the Korean national science curriculum, encourages creating a prepared teaching plan; using various and effective 
materials such as technological materials, science books, models, audio-visual materials, software, internet materi- 
als; and using various and effective teaching strategies such as mutual cooperation, discussion and debates, open 
questions, demonstrations, and so on. 
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Assessment 


Assessment allows teachers to check whether or not teaching and learning goals has been achieved. In addi- 
tion, the Korean national science curriculum emphasizes assessment, including what to and how to assess science 
teaching and learning. The KTOP includes only formative assessments conducted at the end of a lesson because 
such a structure focuses on one-hour teaching and learning in the classroom or laboratory. 


Validity and reliability of the KTOP 
Validity 
For high validity of the KTOP 


As mentioned earlier, to determine each item of the KTOP the following four main sources were referred: actual 
observation of science classes and discussions by five researchers about which aspects needed to be improved, the 
Korean national science curriculum to consider the basic direction and goals of science teaching and learning in 
Korea, literature review regarding more authentic and good science teaching, and existing instruments to measure 
science class, including RTOP and OTOP. 

Through the direct observation of actual science teaching, 12 items were extracted as basic elements of 
the initial version of the KTOP. And after the review process, the initial version of KTOP was revised into 26 items. 
Finally, this revised version was applied in actual science teaching again, and then finalized to 30 items of 4 main 
and 10 sub-categories. 


Content validity 


For obtaining the content validity of the tool, Lawshe’s (1975) method with a few revisions was used. We asked 
48 science teachers to determine whether each item of the KTOP is important for his/her good science teaching 
or not. In the survey question, a description about the meaning of each item of the KTOP was provided to help 
teachers determine the importance of each item. The teachers gave us responses based on the Likert-scale; that 
is, they evaluated an item's importance on a five-point scale of 5, 4, 3, 2, and 1. The value of “5” corresponded to 
“strong agreement,’ “3” a neutral answer, and “1” to “strong disagreement.’ The result (Table 5) showed the high 
validity of the KTOP because the average of the responses was 4.4 for all items, 4.2~4.6 for each category, and 
4.0~4.6 for each item of the KTOP. 


Table 5. Teachers’ responses about the importance of each item of the KTOP. 


Item category 1 2 3 4 

Item 1-1 1-2 #13 #2 314 «432 32 34 35 36 41 #42 #43 44 #45 46 4-7 
Response 44 45 45 44 46 46 46 43 42 43 43 46 44 46 45 45 43 

Average 4.4 44 44 4.4 
Item category 5 6 7 8 9 10 
Item 5-1 5-2 5-3 6 7-1 7-2 8-1 8-2 8-3 9-1 9-2 9-3 10 
Response 45 45 4.5 46 44 44 4.2 4.3 4.0 4.2 44 4.6 4.5 
Average 4.5 4.6 4.4 4.2 4.4 4.5 
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Construct validity 


For construct validity, the operational definition of each item in the KTOP was developed to increase the degree 
to which the KTOP measures what it claims or purports. For convergent validity, concrete examples supporting the 
operational definition of each item were developed. Lastly, for discriminant validity, counter-examples refuting it 
were also developed (as shown in Table 1). In this study, though statistical data, such as factor analysis data, were 
not obtained, it was expected that the operational definition, examples, and non-examples for each item of the 
KTOP can increase the construct validity. 


Reliability 
Agreement between raters 


Reliability can be obtained by agreement among two or more raters. To do this, Cohen’s kappa is usually used 
for two raters, and Light’s kappa or Fleiss’s kappa is used for more than three raters (Hallgren, 2012). However, if 
the rates are inclined toward a one side score, such as a positive or negative score, a kappa value cannot be used 
because of the so-called “kappa paradox” (Kundel & Polansky, 2003; Viera & Garett, 2005). In our case, if science 
teaching is well-implemented, then almost all scores of the KTOP would be positive, or the opposite could be true. 
In this case, the rate of agreement using the formula below is more appropriate than the kappa value (Cicchetti & 
Feinstein, 1990). For more than three raters, we can use the average of the agreements obtained from all possible 
pairs of raters. 


| Number of items being rated samely by two raters 
Rate of Agreement = 2—2—HY TT 
Total number of items to be rated 


To obtain the agreement between raters, an in-service program was developed and implemented. In this 
program, seven physics teachers rated two physics classes and ten earth science teachers rated two earth science 
classes. In the early stage of an in-service program, that is, before acquiring sufficient expert skills to use the KTOP, 
the averages of agreements between teachers for two science classes was 0.67. However, in the later stage, the 
averages of agreement for the other two science classes increased to 0.72~0.90. The averages of agreements be- 
tween a researcher and teachers were 0.54~0.57 in the early stage, but 0.72~0.88 in the later stage of the in-service 
course. From these results, it was observed that, after teachers drilled the criteria for using the KTOP, the agreements 
between raters were high, ranging from 72 to 90%. The detailed information about this in-service program can be 
found in another paper (Jeong, Park, Park, Kim, & Park, 2014). 


Correlation between four observation data 


Swada etal. (2002) used the correlation between observers’ ratings to estimate inter-rater reliability. The same 
method was used in this study. In the in-service program mentioned above, seven physics teachers and one of 
our research teams observed and analyzed two middle school physics classes, and ten earth science teachers and 
another of our team observed and analyzed two middle school earth science classes. From this, four sets of data 
were obtained: average score of seven physics teachers and the score of the researcher for each physics class; 
average score of ten earth science teachers and the score of the researcher for each earth science class (Figure 2). 
From these data, the correlation coefficient was found to be 0.90 (p=0.096), and the variance shared between the 
researcher and teachers was 82%. This high correlation means high inter-rater reliability, even though the number 
of data is too small for a robust analysis at this point. 
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Figure 2: Correlation between researcher and teachers (r = 0.96, p = 0.096, r-squared = 0.82). 


Application of the KTOP 
Applying to POCOM (Practical On-site Cooperation Model) 


The literature notes a gap between educational theory and teaching practice in schools (De Corte, 2000). That 
is, even though there have been significant advances in educational theories and research, and though teachers 
are familiar with many of them, actual classroom teaching has not shown a large improvement. In fact, Park, Park, 
Kim, Jeong, & Park (2014a) also found that Korean science teachers, in their actual science teaching in schools, used 
only 26% of what they know about educational theories and teaching strategies, even though they agree that the 
educational theories and teaching strategies are important for their actual science teaching. 

As mentioned earlier, to become a public science teacher in Korea, students graduating from the college of 
education must pass a very competitive national recruitment examination. Because teaching is considered as a 
favored job in Korea, highly qualified high school students go to colleges of education, which are considered de- 
manding. Therefore, it can be assumed that the level of Korean science teachers’ existing PCK and teaching skills 
are high; the reason for the gap between knowledge and practice may be because such knowledge about the 
PCK and teaching skills are not activated in actual teaching in schools. Therefore, to activate teachers’ existing but 
hidden professional knowledge, the POCOM (Practical On-Site Cooperation Model) was developed. 

In POCoM, researchers observe science teaching directly and use the KTOP to analyze which aspects need to 
be improved. Then, teacher and researcher cooperate to discuss ideas and methods to improve the KTOP items 
that need improvement. The improvement ideas and methods are then directly applied to the next teaching. The 
cyclical process of “observation-analysis-cooperation-application” is continued four times. 

As a result, an improvement rate of about 85% on average could be obtained from three teachers (three 
classroom teaching). When the POCoM was re-applied to the same teachers 1~2 months later, the number of KTOP 
items needing improvement decreased with statistical significance. Therefore, the effectiveness of the POCOoM in 
improving actual science teaching was maintained after the first cyclic application of the POCoM. More detailed 
information about the POCoM application using the KTOP can be obtained in another paper (Park, Park, Kim, 
Jeong, & Park, 2014b). 


Applying to an In-service Program 
In the original POCoM, the observed teaching using the KTOP was analyzed by researchers. However, teachers 
can also analyze his/her own teaching or other teacher’s teaching, if the teacher can use the KTOP professionally. 


To foster this ability, a ten-hour in-service program was developed and implemented for 17 science teachers. 
In this program, the basic features of the KTOP were introduced first, and then the participants analyzed the first 
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video-recorded classroom teaching conducted by another teacher. After the analysis, the participants presented, 
compared and discussed their analysis results. And they cooperated with each other to produce ideas to improve 
the KTOP action items in the first video. Participants observed and analyzed a second video and compared their 
analysis results of the first and the second video.. 

As noted in reliability section, agreements between teachers were 0.67~0.67 for the first video and 0.72~0.90 
for the second video. Agreements between a researcher and a teacher were 0.54~0.57 for the first video and 
0.72~0.88 for the second video. From this data, we concluded that the teachers’ analysis expertise was increased 
through the in-service program. More detailed information about this study will be published in the near future 
(Jeong et al., 2014). 


Transforming the KTOP to Different Versions 


The original version of the KTOP consisted of 4 main and 10 sub-categories with 30 items. However, this 
number of items was too large to be checked during science teaching in real time. In fact, according to our study 
of applying the KTOP to POCoM (Park et al., 2014b), the average number of KTOP items that needed improvement 
was 11 on average in three teaching classes. 

Therefore, to increase the convenience in using the KTOP in real time during a class observation, the num- 
ber of items of the KTOP should be decreased. However, because all items included in the original version of the 
KTOP are important for science class evaluation, we cannot exclude any item arbitrarily. In this case, more POCOM 
studies are necessary to obtain data indicating which items are most checked or least checked. For example, in 
the study of Park et al. (2014b), the most checked items as needing to be improved in three science classes were 
9-3; then 2, 1-1, 6, 9-1, and 9-2; then 1-3 and 3-3; then 3-4; then 8.3, and so on. A smaller version of the KTOP can 
be suggested with these ten items (9.3, 2, 1-1...). Of course, to develop this more compact instrument, more data 
should be obtained from more applications of the KTOP to the POCoM. 

In addition, using many observed results of one science teacher will help identify the most checked items 
for that particular teacher. Then, another compact version of the KTOP can be suggested, that is, a“Personal KTOP 
version.” However, these transformed versions of the KTOP have yet to be applied. Therefore, we expect to conduct 
further studies regarding this point. 


Conclusions 


In this study, the KTOP for identifying which aspects need to be improved in science teaching was developed 
with the following four features: (1) The developmental process of the KTOP was designed to diminish the gap 
between theory and practice using the bottom-up (direct observation of teaching at school) and top-down ap- 
proaches (literature review, consideration of Korean national science curriculum, and comparing it with RTOP and 
OTOP). (2) It is specialized for science teaching; therefore, it can be used for improving science laboratory teaching 
as well as classroom teaching. (3) It is developed for improving science teaching in a more practical way as well 
as for evaluating it. (4) To do this, it includes concrete examples (which indicate aspects to be encouraged) and 
counter-examples (which indicate aspects to be improved) developed based on actual observations of science 
teaching. 

As mentioned above, when developing the KTOP, the Korean national science curriculum was considered 
because it defines basic goals of science education, science contents to be learned in each grade, teaching plans, 
guides for assessment, and so on. In fact, the Korean national science curriculum has been constructed and reformed 
considering the curricula of other countries, including ‘science for all Americans (American Association for the 
Advancement of Science, 1993; 1994), ‘science education standard’ (National Research Council [NRC], 1996; 2000), 
and ‘a framework for k-12 science education’ (NRC, 2012) in the US, and ‘four key stages’ of the national curriculum 
in the UK (Department of Education, 2013). Therefore, even though the KTOP was developed based on the Korean 
curriculum, the KTOP can also be utilized in other countries without any changes in each item of the KTOP. 

Of course, any single educational theory or tool cannot be applied to all teaching contexts (Loughran et al., 
2006; Trianou, 2007) because teaching is context-based. This means that we cannot confirm that the KTOP can be 
used in other educational contexts in other countries in its original form. Therefore, some items of the KTOP can be 
revised according to the specific educational situation; then, the KTOP may be renamed to “Revised-ATOP” (here, 
‘A’ denotes the initial of the name of the country). 


2/2 


WSs 


Journal of Baltic Science Education, Vol. 13, No. 2, 2014 


ISSN 1648-3898 THE DEVELOPMENT OF THE KOREAN TEACHING OBSERVATION PROTOCOL (KTOP) FOR IMPROVING 
SCIENCE TEACHING AND LEARNING 
(P. 259-275) 


The KTOP was developed for improving science teaching through observing and analyzing the science class, 
however, it can be used for other purposes. For instance, it can be applied to develop teachers’ expertise in sci- 
ence teaching through pre and in-service teacher program, to examine and to elicit the characteristics of science 
instruction in different teaching situations, and also can be utilized as an evaluation tool for selecting the capable 
teachers in the process of teacher recruitment examination, and so on. Thus, we trust that the KTOP can be actively 
employed in further works. 
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